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Abstract 

We investigate the nonlinear instability of a smooth steady density profile solution of the three- 
dimensional nonhomogeneous incompressible Navier-Stokes equations in the presence of a uniform 
gravitational field, including a Rayleigh- Taylor steady-state solution with heavier density with 
increasing height (referred to the Rayleigh- Taylor instability). We first analyze the equations ob- 
tained from linearization around the steady density profile solution. Then we construct solutions 
of the linearized problem that grow in time in the Sobolev space H k , thus leading to a global 
instability result for the linearized problem. With the help of the constructed unstable solutions 
and an existence theorem of classical solutions to the original nonlinear equations, we can then 
demonstrate the instability of the nonlinear problem in some sense. Our analysis shows that the 
third component of the velocity already induces the instability, this is different from the previous 
known results. 

Keywords: Nonhomogeneous Navier-Stokes equations, steady density profile, Rayleigh- Taylor 
instability, incompressible viscous flows. 
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1. Introduction 

This paper is concerned with the nonlinear instability of a smooth steady density profile 
solution of the three-dimensional nonhomogeneous incompressible Navier-Stokes equations in 
the presence of a uniform gravitational field, including a Rayleigh- Taylor steady-state solution 
with heavier density with increasing height (referred to the Rayleigh- Taylor instability). 

The motion of a nonhomogeneous incompressible viscous fluid in the presence of a uniform 
gravitational field in R 3 is governed by the Navier-Stokes equations: 

pt + v ■ Vp = 0, 

pw t + pv ■ Vv + Vp = pAv - pge 3 , (1.1) 
divv = 0, 

where the unknowns p, v and p denote the density, the velocity, and the pressure of the fluid, 
respectively. In the system (11. ip we have written p > for the coefficient of shear viscosity, 
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g > for the gravitational constant, e% = (0, 0, 1) for the vertical unit vector, and — ge 3 for the 
gravitational force. 

In this paper we consider the problem of the Ray leigh- Taylor (RT) instability, so we assume 
that a smooth steady density profile p := p(x 3 ) G C°°(R) exists which satisfies 

p' G C °°(1R), inf p > 0, (1.2) 

X3&R 

p'{xf) > for some point x° G R, (1.3) 

where ' = d/dx%, see Remark 11.41 on the construction of such p. Clearly, such p with v(£, x) = 
defines a steady state to (II. ip . provided 

dp 

yp = -pgez, i-e., -j— = (1.4) 

<±r 3 



Remark 1.1. We point out that by virtue of the condition ( ll.3|) . there is at least a region in 
which the steady density solution has larger density with increasing x% (height), thus this will 
lead to the classical Rayleigh- Taylor instability as shown in Theorem 11.11 below . 

Let the perturbation be 

g = p-p, u = v + 0, q=p-p, 
then, (g, u, q) satisfies the perturbed equations 

g t + u ■ V(g + p) = 0, 

(g + p)u t + (g + p)u ■ Vu + Vg + gge 3 = pAu, (1.5) 
divu = 0. 

To complete the statement of the perturbed problem, we specify the initial and boundary 
conditions: 

(g, u)| t=0 = (g , u ) inR 3 (1.6) 

and 

lim u(t,x) = foranyt>0. (1.7) 

|x|— >+oo 

Moreover, the initial data should satisfy divuo = 0. 

If we linearize the equations ( 11.5|) around the steady state (p, 0), then the resulting linearized 
equations read as 

{Qt + p'u 3 = 0, 
pu t + Vq + gge 3 = pAu, (1.8) 
divu = 0. 

It has been known for over a century that the steady states (p, 0) to the linearized RT 
problem (11.61) — (11.8)) with p > is unstable [H, [l^], i.e., there exists a unstable solution to 
(ll.6l) - (ll.8l) . Such instability to (ll.6p - (ll.8l) is often called linear RT instability. However, there 
have been only few results on the mathematically rigorous justification of the RT instability for 
(jra-flOl. In 2003, Hwang and Guo [l2| proved the nonlinear RT instability to (fT3]) -([TT6l) 
with boundary condition u • n|^ = for the two-dimensional inviscid case (i.e. p = 0) where 
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Q = {(x\,X2) G M 2 I — I < X2 < m} and n denotes the outer normal vector to dQ. To our 
best knowledge, however, it is still open mathematically whether there exists a unstable solution 
to the nonlinear RT problem fll.5p - fll.7l) of viscous fluids with variable density. The aim of this 
article is to show rigorously the instability for the nonlinear RT problem fll.5l) - fll.7p in some 
proper sense. The main result read as follows. 

Theorem 1.1. Let the steady density profile p satisfy / ti-j^ -z TOj) . Then, the steady state (p, 0) 
of U.5\) - [T71\ ) is unstable under the Lipschitz structure, that is, for any s > 2, 5 > 0, K > 0, 
and F satisfying 

F(y) < Ky for any y G [0, oo), (1.9) 
there exist a constant i$ := io(s) > and smooth initial data 

(eo,uo) G (-£/~°°(lR 3 )) 4 with ||(2o, u )||h» ( r3) < 5, 

but the unique classical solution (g,u) of lil.5\) - [T77j) , emanating from the initial data {qo,Uq), 
satisfies 
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\ua(tK)\\i?Qp) > F (W(So, Uo)||h»(r3)) for some t K G ( 0, ^ ln — 



c(0,T max ), (1.10) 



where the constant A is given by \2. 54}) , H°°(M?) = r\^ =1 H k (M. 3 ) , and T max denotes the maximal 
time of existence of the solution (g, u). 

Remark 1.2. It should be noted that we can not obtain the same instability result for the 
domain Qf 1 :={xGlR 3 | — / < £3 < m} in place of M. 3 , due to lack of an existence result of the 
classical solution to the RT problem f ll.5p in the domain VL™. We also mention that Theorem ll.il 
still holds if we define ||(^ , u )||^ s(R3) := HeolHr-i^s) + ll u o||^(R3)- 

Remark 1.3. Our result shows that the problem (11 .5fl — fjl .7p does not possess the following 
stability structure: 

3 a constant C > 0, such that sup ||w3(t)||L2( R 3) < C||(goj u o)||h s (r 3 ) for any T > 0, (1.11) 

te(o,T] 

which should be quite general and reasonable for a global stability theory. Notice that s > 2 in 
Theorem 11.11 is arbitrary. Thus, even if the initial data of the f ll.5l) - fll.7p are smooth and small, 
the failure of the stability structure (11. lip means that it is not possible to use ( 11.111) to control 
the norm of ||u(t) ||z, 2 (m 3 ) for long time. 



Remark 1.4. Here we give an example of a steady density profile solution satisfying the condi- 
tions of Theorem 11.11 Assume 

p h for x 3 > 1, 

P a ={ (p h + P l )/2 for S3 G (-1,1), 
p l for x 3 < —1, 

and < p l < p h < +00, then p := S £ (p a ) and p := g p(s)ds satisfy fll.2D - fll.4p . where S £ is a 
standard mollifier operator. 
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Remark 1.5. The constant A (in (I2.54p ) is often called maximal linear growth rate. By virtue 
of ( I2.54p and (12. 29 p . A < oo, and A — > if g\\P / p\\l°°{r) — > or ji — > oo. In contrast, A = oo in 
the corresponding inviscid case. This clearly shows that the viscosity plays an stabilizing role in 
the linear RT instability. 

The proof of Theorem 11.11 inspired by jsl, [loj], is divided into four steps: (i) First we notice 
that the coefficients in the linearized equations (II .8p depend only on the vertical variable 13 6 R 3 , 
this allows us to seek "normal mode" solutions by taking the horizontal Fourier transform of the 
equations and assuming the solutions grow exponentially in time by the factor e A ^'- )t , where 
£ G M 2 is the horizontal spatial frequency and A(|£|) > 0. This reduces the equations to a system 
of ordinary differential equations (ODEs) defined on M. with A(|£|) > for each £. Then, solving 
this ODE system by a modified variational method, we can show that A(|£|) > is continuous 
function on (0, 00), and the normal modes with spatial frequency grow in time, providing thus 
a mechanism for the global linear RT instability. Consequently, we form a Fourier synthesis of 
the normal mode solutions constructed for each spatial frequency £ to construct solutions of the 
linearized equations that grow in time, when measured in H k (JBL 3 ) for any k > 0. This is the 
content of Section 2. (ii) In Section 3, we show a uniqueness result of the linearized problem (see 
Theorem 13. ip in the sense of strong solutions. In spite of the uniqueness, the linearized problem 
is global unstable in H k (M?) for any k. (iii) Then we derive some nonlinear energy estimates 
of the perturbed problem with small initial data, which make it possible to take to the limit in 
the scaled perturbed problem to obtain the corresponding linearized equations, (iv) Finally, in 
Section 5, with the help of the results established in Sections 2-4 and the Lipschitz structure of 
F, we can obtain the instability of the nonlinear problem in the sense of (ll.lOp . In the proof, 
we shall see that the stability structure (II. lip would give rise to certain estimates of solutions to 
the linearized problem ( 1 1 . Cj j) — ( TTTH|) that cannot hold in general because of Theorem 12.11 

We should point out that the RT instability based on the Lipschitz structure was studied 
by Guo and Tice in [9J for compressible inviscid fluids, where the instability was shown in the 
if 3 -norm of (g, u) and the flow map (see (I5.12p in [9]). Our instability result Theorem 11.11 differs 
from that of Guo and Tice in that only HttaHp is needed here to describe the instability. This is 
also different from that of Guo and Hwang (l2|, in which the instability for an inhomogeneous 
incompressible inviscid fluid is described by the norm ||(^, u)||^2(q). Roughly speaking, our in- 
stability in terms of || 1| z, 2 only is based on two important observations: (i) one can construct a 
solution (g, u) with ||w3(0)||i2 > of the linearized problem; (ii) more regularity of the solution 
(g, u) to the corresponding nonlinear problem can be derived from the problem (ll.5p - (ll.7p . we 
refer to Section 5 for details. We also mention that in the current paper we have to employ 
new techniques to construct growing in time solutions to the linearized problem. To construct 
such solutions, we shall first transform the linearized equations to an ODE system. In the the 
inviscid fluid case, the ODE system can be viewed as an eigenvalue problem with eigenvalue —A 2 



(cf. ODE (10) in [12J for impressible fluids, or ODE system (3.11) in |9J for compressible fluids). 
Unfortunately, when the viscosity is present, the linear term multiplied by A breaks down the 
natural variational structure, such that the variational method can not be used. In order to 



circumvent this problem, for compressible viscous fluids, recently Guo and Tice [lOj artificially 



removed the linear dependence on A by first defining s := A > 0, then solving the family of 



modified problems for each s > 0, and finally showing s = X(s) for some s. In [lOj the ODE 
system is defined in a bounded domain, and the compact imbedding, an important step in their 
construction, can thus be applied. In our case, however, our ODE (see (12. 4p ) is defined on R, and 
consequently the compact embedding does not hold. To overcome this difficult, here we exploit 
the property of weak convergence and the structure of the energy functional E(ip) corresponding 
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to our ODE. In particular, we develop a new analysis technique to prove A(|£|) G C°(0,oo) by 
first showing A(|£|, s) G C°(0, oo) for each fixed s, and then exploiting the monotonicity of A(-, s) 
to further verify A(|£|) := A(|£|, A) G C°(0, oo), see the proof of Proposition 12.51 for details. 

We end this section by briefly reviewing some of the previous results on the nonlinear RT 
instability for two layer incompressible fluids with a free interface, where the RT steady state 
solution is a denser fluid lying above a lighter one separated by a free interface. When the densities 
of two layer fluids are two constants, Priiess and Simonett [18] used the C°-semigroup theory and 
the Henry instability theorem to show the (local) existence of nonlinear unstable solutions in the 
Sobolev-Slobodeckii spaces, where the instability term is described by the sum of ||u|| 2-2/p and 



h|| w 3-2/p (see [18|, Theorem 1.2] for details). When densities of two layer fluids are variable, to 



our best knowledge, the (local) existence of solutions to the nonlinear problem is still not known 
unfortunately, and thus the nonlinear instability is still open. For compressible fluids there are 
very few results on the nonlinear RT instability. Guo and Tice proved the instability of immiscible 
compressible inviscid fluids in the frame of Lagrangian coordinates under the assumption of the 
existence of solutions [9(, which is in some sense a compressible analogue to the local ill-posedness 
of the RT problem for incompressible fluids given in Q. Recently, Jiang, Jiang and Wang 



151 ] adapted Guo and Tice's approach to investigate the nonlinear instability of two immiscible 
incompressible fluids with or without surface tension in Eulerian coordinates without the help of 
a coordinate transformation. We remark that the analogue of the RT instability arises when the 
fluids are electrically conducting and a magnetic field is present, and the growth of the instability 
will be influenced by the magnetic field due to the generated electromagnetic induction and the 
Lorentz force. Some authors have extended the partial results concerning the RT instability of 
superposed flows to the case of MHD flows by overcoming the more complicated structure due 
to presence of the magnetic field, see 0, HH, 14, 16, 24 . 

Notation: Throughout this article we shall repeatedly use the abbreviations: 

W m ' p := W m ' p (R 3 ), H m := H m (R 3 ), U> := L P {R 3 ), 

etc. 



|| • ||W"»,j> •— || ' ||W m 'P(IR 3 ); II ' \\H m ■— || ' \\H m (R 3 ), || ' 1 1 LP •— II ' \\Lp 

2. Construction of solutions to the linearized problem 

We wish to construct a solution to the linearized equations fll.Sp that has growing if fe -norm 
for any k. We will construct such solutions via Fourier synthesis by first constructing a growing 
mode for any but fixed spatial frequency. 

2.1. Linear growing modes 

To begin, we make a growing mode ansatz of solutions, i.e., 

£>(x) = p(x)e At , u(x) = v(x)e A ', g(x) = p(x)e A< for some A > 0. 

Substituting this ansatz into (jl.8|) . and then eliminating p by using the first equation, we arrive 
at the time-invariant system for v = (vi, V2, v 3 ) and p: 

A 2 pv + AVp = ApAv + gp'v 3 e 3 , ^ 



div v = 



with 



lim v(x) = 0. 

|x|— »+oo 
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We fix a spatial frequency £ = (£1,^2) £ ^ 2 and take the horizontal Fourier transform of 
(vi,v 2 , v 3 ) in (12.11) . which we denote with either : or J 7 , i.e., 

fe,x S )= [ f(x',x 3 )e- ix 'Mx'. 



Define the new unknowns 

(p(x 3 ) = ivi(i, ^3), 0(x 3 ) = zt> 2 (f , x 3 ), ip(x 3 ) = u 3 (f , x 3 ), tt(x 3 ) = p(f , x 3 ), 

so that 

^(divv) = £np + i 2 6 + if;', 
where ' = d/dx 3 . Then, for (p, 8, ip and A = A(£) we arrive at the following system of ODEs 

' A 2 ^-A£ 1 7r + A yU (|£|V-^") = 0, 
\ 2 p6 - X&it + X^^O - 6") = 0, 
A 2 p^ + Avr' + AM|£| 2 ^ - = gp'ip, 



(2.2) 



with 



(2.3) 



(f( — OO) = 6 I ( — OO) = lj)( — Oo) = (y9( + 00) = #( + 00) = ^( + 00) = 0. 

Eliminating 7r from the third equation in (I2.2p we obtain the following ODE for ij) 

- a 2 [pKI 2 ^ - G#')1 = AM|£| 4 v - 21^1 V + iT) - <?i£i 2 pV 

with 

ip(-oo) = ip'(-oo) = i()(+oo) = ^'(+00) = 0. 

Similarly to [10], we can apply the variational method to construct a solutions of (I2.4l) -( f2~5l) . 
The idea of the proof can be found in Guo and Tice's paper for viscous compressible flows 



(2.4) 



(2.5) 



10 1 , and was adapted later by other authors to investigate the instability for other fluid models 

JT3, 0,3- 

Now we fix a non-zero vector £ e K 2 and s > 0. From (12.4p - (12.5l) we get a family of the 
modified problems 



coupled with (12.51) . We define the energy functional of (12.61) by 

ew= [ S /i(4iei 2 i^ 2 +iieiv+<i 2 )-5ieiWdx 3 



m-= / p(ieri^i 2 + i^r)dx 3 = 1 



with a associated admissible set 

A={ipe H 2 ( 



(2.6) 



(2.7) 



(2- 



Thus we can find a —A 2 by minimizing 



A 2 (|£l) = «(|£|):= 



inf EN>). 



(2.9) 



In order to emphasize the dependence on s £ (0, oo) we will sometimes write 



E(if>, s) := E(ip) and a(s) := inf E(if>, s). 

ip€A 



Next we show that a minimizer of (I2.9P exists for the case of mi^E(ip, s) < 0, and that the 
corresponding Euler-Lagrange equations are equivalent to (12.51) . (12.61) . 

Proposition 2.1. For any fixed I; with |£| 7^ 0, inf^ e _4 E(tJ), s) > —00. In particular, if there 
exists a ip £ A, such that E(ip) < 0, then E achieves its infinimum on A. In addition, let ip be a 
minimizer and —A 2 := E{jp), then the pair (ip, X 2 ) satisfies Ii2.5\) . 112.6}) . Moreover, ip £ H k (R) 
for any positive integer k. 



PROOF. We first note that for any ip £ A, 

E(4>) > -<7|£| 2 / pV 2 dx 3 > -g P - [ 



dx 3 > -g 



(2.10) 



Hence E is bounded from below on A by virtue of (II .2p . Let ip n £ A be a minimizing sequence, 
then E(ip n ) is bounded. This together with (12 .8p and (12. 7p again implies that ip n is bounded 
in H 2 (R). So, there exists a ip £ H 2 (M), such that tp n — > ip weakly in H 2 (M.) and strongly 
in H} oc (R). Moreover, by the lower semi-continuity, locally strong convergence, (ll.2p and the 
assumption that E(ip) < for some ip £ A, we have 

E(i/j) < liminf E(i/; n ) = inf E < 0, and < J(ip) < 1. 

n— >oo A 

Suppose by contradiction that J(ip) < 1. By the homogeneity of J we may find an a > 1 so 
that J(aip) = 1, i.e., we may scale up ip so that at]) £ A. From this we deduce that 

E(aaj)) = a 2 E(ip) < a 2 inf E < inf E < 0, 

which is a contradiction since aip £ A. Hence J{ip) = 1 so that ip £ A. This shows that E 
achieves its infinimum on A. 

Notice that since E and J are homogeneous of degree 2, (12. 9p is equivalent to 



a(s) 



inf 



(2-11) 



For any r £ IR and ipo £ i7 2 (K) we take V'( r ) = "0 + r ^o ; then (12.111) implies 

E(iP(t)) + X 2 J(V»(r)) > 0. 

If we set I(r) = E(i/)(t)) + A 2 JY>(r)), then we see that 7(r) > for all r £ R and 7(0) = 0. 
This implies I'(0) = 0. By virtue of (1 2 .7ft and (I2.8p . a direct computation leads to 



(2.12) 



si* / (4iei Wo + (lei v + Oder^o + 

= ^| 2 / pVVod^-A 2 / p(|^| 2 ^o + ^' )dx 3 , 



where we have used the upper boundedness of p. 
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By further assuming that ip is compactly supported in R, we find that ip satisfies the equation 
( 12. 6 p in the weak sense on R. In order to improve the regularity of if), we rewrite f !2.12p as 

(213) 

f?Po<lx 3 . 

For any n > 1, let Vi.nj^ £ QTdO satisfy ^i, n (#3) = 1 for |x 3 | < n. If we take ipo — 
ipi, n f_ 3 V^dr in (I2.13p . then we have 



.'•3 



{M'^dxa = I [ M lB / ^ 2 dr - / ^dr - 2^i,» 2 dz 3 

i-oo J —oo J 

which, recalling ^ G if 2 (R), implies ■?/>" G ii/j^R) and 

/•+00 

■0'" = (VV0")' = / (/V'l.n - 0i n^")dT for any x 3 with |x 3 | < n. 



Integrating by parts, we can rewrite (I2.13P as 

r%dx 3 = - f {g\i\ 2 pi, - \ 2 (\e^ - wy) + *M2iei v - ieiV)) ^d* 3 , 

S A* JR 

which, keeping in mind that V e # 2 (R), yields ip"" G L 2 (R). Hence ip G i^ c (R) fl C^ 2 (R) , and 
ip (oo) = ip (oo) = ^ (oo) = 0. Using these facts, Holder's inequality, and integration by parts, 
we deduce that 

= [ \r\ 2 dx 3 = - [ rr"dx 3 < hvi^iivh^w, (2.14) 

Jr Jr 

i.e., if)'" G L 2 (R). Consequently, if) G iJ 4 (R) and solves (I2.5p - fl2.6p . This immediately gives that 
if) G H k (R) for any positive integer k > 5. □ 

Next, we want to show that there is a fixed point such that X = s. To this end, we first give 
some properties of a(s) as a function of s > 0. 

Proposition 2.2. The function a(s) defined on (0, oo) enjoys the following properties: 

(1) For any a, b G (0, oo) with a < b, there exist constants c\, c 2 > depending on p, p, g, a 
and b, such that 

a(s)<-c 1 + sc 2 for all |£| G [a, b\. (2.15) 

(2) a(s) G C lo ' c (0, oo) is nondecreasing. 

Proof. (1) In view of ffOj) . there exists a ip G C£°(R) such that 

#a 2 / R pV 2 dx 3 



/ R p(6 2 H 2 + |V/| 2 )d* 3 



:= d > 0, (2.16) 



where the constant C\ depends on a, b, g and p. Now, we use ( 12.111) and (I2.16P to find that 



as 



< s 



^ 2 (M) J(V>) " / R P(|f PM* + i^i 2 )dx 3 

^/ R (4K| 2 |^| 2 + IK| 2 ^ + ^| 2 )dx 3 ^a 2 / M pV 2 dx 3 



2LAI2 



|^'| 2 )da;3 



J R p(6 2 |^| 2 + |^| 2 )dx 3 



sc 2 - Ci, 



where the positive constant c 2 depends on p, p, g>, a and b. Hence, (I2.15P holds. 

(2) To show the second assertion, we let Q :— [a, b] C R be a bounded interval, and 



e 1 w = p / m ^r + \\^+r\ 2 )dx 3 . 

Jr 

For any s G Q, there exists a minimizing sequence {ip™} C ^4 of mi^ e ^E(ip, s), such that 

\a(s)-E(^,s)\<l. (2.17) 
Making use of (12771) . (127T01 . (127151) and (127171) . we infer that 

EaW^s) = : 1 — / pijj dx 3 



< 



s s 
l + max{|&c 2 -ci|,#||p7p|| ioo(R) } g 



(2.18) 



K. 



For Si G Q {i = 1, 2), we find that 

a(si) < limsup -E(<p™, Si) < limsup E{^, s 2 ) + \si - s 2 \ limsup £aC0™ 2 

n— >co n— >oo n— >oo 

<a(s 2 ) + K\si - s 2 |, 



(2.19) 



where {V^} C .4. is a minimizing sequence of inf^, g _4 E(i/j, s 2 ) and the constant if is given in 
( I2.18p . Reversing the role of the indices 1 and 2 in the derivation of the inequality (|2.19p . we 
obtain the same boundedness with the indices switched. Therefore, we deduce that 

|a(si) - a(s 2 )\ < K\si - s 2 |, 

which yields a(s) G C lo ' c (0, oo). 

Finally, from ([277]) and fl279]) it follows that 

^(si) < limsup E(i/j™, s\) < limsup E(ijj™ , s 2 ) = a(s 2 ) for any < s% < s 2 < oo. 

n—^oo n— ¥oo 

Hence a(s) is nondecreasing on (0, oo). This completes the proof of Proposition 12.21 



□ 

Given £ G M 2 with |£| ^ 0, by virtue of (I2.15p . there exists a s > depending on the 
quantities p, p, g, |£|, such that for any s < s , a(s) < 0. Let 



(3 If I := sup{s | a(r) < for any r G (0, s)} > 0, 



(2.20) 



then @|f | > 0. This allows us to define A(s) = yj— a(s) > for any s G <S|f| := (0, ©|f|). Therefore, 
as a result of Proposition 12.11 we have the following existence for the modified problem ( 12. 6p . 

(HZD- 
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Proposition 2.3. For each |£| 7^ and s G Sm there is a solution ip = 4>{\£,\, x 3) ¥ with 
A = A(|£|, s) > to the problem \2. 5\) . A2.6}) . Moreover, ip G H k (R) for any positive integer k. 

Now, we can use Proposition E21 fl2T20l) and f[2~T0D to check that A(s) G C^ L C (S\ & ) is nomn- 
creasing (in fact, we can further show that X(s) is strictly increasing, we refer to the proof of 



10, Proposition 3.6]), A(s) < s /g\\y/p'/ p||l°°(r), and lim^^e^ A(6|^|) = if &\$\ < +00. Hence, 
we can employ a fixed-point argument to find s G S\£\ so that s = A(|£|,s), thus and obtain a 
solution to the original problem (12.41) . (12.51) . 



Proposition 2.4. Let |£| 7^0, t/ien there exists a unique s G <Si£i, swc/i t/iai A(|£|, s) = y— a(s) > 
and s = A(|£|, s). 



Proof. We refer to 10|, Theorem 3.8] (or [24J, Lemma 3.7]) for a proof. □ 



Consequently, in view of Propositions 12.31 and 12.4} we conclude the following existence for the 
problem (l^ijl . fl2T5|) . 



Theorem 2.1. For eac/i |£| 7^ 0, there exist ip = -0(|£|, £3) ^0 and A(|£|) > satisfying \2.J$ , 
j[2.5\) . Moreover, ip G H k (R) for any positive integer k. 

We end this subsection by giving some properties of the solutions established in Theorem 12.11 
in terms of A(|£|), which show that A is a bounded, continuous function of |£|. 

Proposition 2.5. The function A : (0, 00) — > (0, 00) is continuous and satisfies 



sup \(\£\)<y/g VP'/P , • (2-21) 

Proof. The boundedness of A (I2.2ip follows from (I2.10p . To show the continuity of A, we see 
that for any but fixed £ 7^ 0, there exists an interval [a, b] C (0, 00) so that |£ | £ (o, b). Assume 
l£l ~* |£o| with |£| G (a, 6), and denote k = |£| 2 - |£ | 2 , then k -> as |£| ->■ |£ |- 
(i) We first show 

lim a(\£\,s) = a(|f |,s) for any s G S\%\. (2.22) 

l£M£o| 

By virtue of Proposition I2.1[ for any |£| G (a, 6), there exists a functions G A, such that 

«(iei)= / ^4ieiMd 2 + ii^i 2 %i+^ii 2 )-^iei 2 4i^3 (2.23) 

J R 

Utilizing (EH]) and (I27L5) . we have 

|jf*(R) < c 5 , (2.24) 



where c 5 depends on p, /z, g, a, b and s. 

Substitution of|£| 2 =|£| 2 + /t into fl2T23|) results in 



a(iei) = / s^(4ie |V| S ir + ne r%i + vg 2 ) - g\u 2 p%dx 3 + 

</R 

>a(|Co|) + «/(«,%|), 

where 

/(«, = / s/x(4|^| | 2 + 2%,(|e| 2 %| + ^|) + «^f c , - g\Z\ 2 p%dx 3 . 
Jr 
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(2.25) 



By Holder's inequality and (I2.24p . we can bound 



\f(K,i[)\£\)\ < Cq for some constant Cq. 



(2.26) 



Similarly to (I2.25f) and (12. 26 p . we also have 



a(ICol) > «(!£!) - K /(- K ,^|? |) and !/(-«> V^fol)! < c 6 



(2.27) 



Combining ( 12T25D with f l2T27D . we get 



«/(-«> V'l&l) ^ - «(lCo|) > «/(k,^|), 



which, together with (T2T2l)j) and (E2ZD, implies that ( I2T2"2"]) . Hence 



lim A(|£|, s) = A(|Co|, s) for any s G <%, 



(2.28) 



because of A(|£|, s) = a/— s). 

(ii) Exploiting (I2.28P and Propositions 12.41 we know that for any e > 0, there exists a 
5 > such that |A(|£|, s^ \) - A(|£ |, < £ and = A(|£o|, S|$ |) = y/~ a (\^ s \^o\) for an y 
I l£l — l£o| I < 8- On the other hand, for each |£| > 0, A(s) is nonincreasing and continuous on 
<S|£|, and there exists a unique s\%\ G <S|£| satisfying A(|£|,S|g|) = S|£| > by Propositions 12.41 
Consequently, we immediately infer that |A(|£|,S|g|) — A(|£o|> s |£ |)| < e with s\£\ = A(|£|,S|£|). 
Hence A(|£|) is continuous. This completes the proof of the proposition. □ 

Remark 2.1. In addition, since p' G Cq°(R), we can bound A as follows, 
(i) Applying integrating by parts and Holder inequality, 





Consequently we have lim^^o A(|£|) = 0. 

(ii) There exists a functions G A such that 




which implies that there exists a constant C(p) depending on p such that 




We immediately get that 



A(|£|)< 



gC(p) 



(2.29) 



4/x 
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2.2. Construction of a solution to the system h2.2\) . A2.3\) 

A solution to (12. 4p . (12. 5p gives rise to a solution of the system (I2.2p . (I2.3P for the growing 
mode velocity u as well. 

Theorem 2.2. For each I; e IR 2 with |£| > 0, there exists a solution (if,9,ip,ir) = (v?(£, x 3 ), #(£, x 3 ), 
il>(\€\,x 3 ),n(\£\,x 3 )) with A = A(|f|) > to / TOP . / TO) , and the solution belongs to (H k (R)) A for 
any positive integer k. 

Proof. With the help of Theorem I2.1[ we first construct a solution (if),\) = C0(|£|, £3), A(|£|)) 
satisfying (E3J), Recalling A > and if) e AnH k (R) for any positive integer k, multiplying 

fl2.2|) i and (12.21) 9 by £1 and £ 2 respectively, adding the resulting equations, and utilizing (j2.2P /i. 
we find that n can be expressed by if), i.e., 

vr = 7r(iei,%) = w - (Ap+/iieiV]ier 2 . (2.30) 

Next, we construct the solution (if, 9). To this end, we shall exploit the fact that the problem 
(12.21) . (12.31) is invariant under simultaneous rotations of (<p,9) and (£i,£ 2 ). Indeed, it is easy to 
see that if 1Z G £0(2) is a rotation operator, then lZ(ip,9), 7£(£i,£ 2 ) is also a solution with the 
same if), tt and A. Thus, given any £ we choose a rotation operator so that = (|£|,0). 
Hence, 

(&M,7T,A)= (-4|(|e|)/|£|,0,V(|ei),7r (given by flZSDD), A(|£|)) 
is a solution to (Q, (Q with (£i,£ 2 ) = (|f|,0). Now, if we define 

(p,M,7r,A) := (H^(<p,9),xP,n,X) = (-(6, k)4|/l£| 2 , V>, ?r, A), (2.31) 

we find that (■0, vr, A) constructed above is indeed a solution to the problem (12. 2p . (12. 3p . □ 

Remark 2.2. For each x 3 , it is easy to see that the solution (<p(£, •), 9(£, -),if;(\^\, •), vr(|£|, •), A(|£|)) 
constructed in Theorem 12.21 has the following properties: 

(1) A(|£|), ip(\£\, •) and 7i"(|£|, •) are even on £1 or £ 2 , when the another variable is fixed; 

(2) <£>(£, ■) is odd on £1, but even on £ 2 , when the another variable is fixed; 

(3) #(£, •) is even on £ 1; but odd on £ 2 , when the another variable is fixed. 

The next lemma provides an estimate for the if fc -norm of the solution (</?, 9, if), 7r) with £ 
varying, which will be useful in the next section when such a solution is integrated in a Fourier 
synthesis. To emphasize the dependence on £, we write these solutions as (<p(£) = y?(£, x 3 ), #(£) = 
0(t,Xs)Mt) = 0(|eU 3 ),7r(O = n(\Z\,x 3 )). 

Lemma 2.1. Let (el 2 with < R x < |£| < R 2 , <p(£), 9(£), 0(£), tt(£) and A(£) be constructed 
as in Theorem \2.2\ then for any k > there exit positive constants A k , B k , C k and D, which 
may depend on R\, R2, p, /i and g, such that 

11^(0 IU*(R) <A k , (2.32) 
H0\\HHR)<B k , (2.33) 

l|y(OIU*W + H (OIU*(R)<Cfc- (2-34) 

Moreover, 

Wi» W >0. (2.35) 
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Proof. Throughout this proof, we denote by c a generic positive constant which may vary from 
line to line, and depend on R 1: R 2 , p, p and g. 

(i) First, since ip G A, we see that f l2.35|) holds, and there exists a constant c, such that 



< c. (2.36) 



On the other hand, in view of Proposition I2.5[ we have 

A(f) > c > for any |f | G (R u R 2 ). (2.37) 
Similarly to f l2TMjl . we use fT2T36D . (1237]) and fl2~23l) with A(f) in place of a(f) to deduce that 

U{0\Wm<c. (2.38) 

We now rewrite (12.41) as 

= [X(Xp + 2p\eW(0 + AW(0 - KI 2 (A 2 P + A/i^l 2 - /A//, (2.39) 

which, together with fl2TT4l) . f[2T3Tj) and fl2T38|) . yields 

l^m < c. (2.40) 



Differentiating (12.391) with respect to 23 and using (12.401) . we find, by induction on k, that 
(I2.32p holds for any k > 0. Consequently, we can employ f!2.32j) and the expression (I2.30j) of it 
to deduce that f!2.33j) holds for any k > 0. 

(ii) Finally, we verify <^M>- By virtue of (|2T3Tj) . 9) = -V>'(£i, 6)/|£| 2 . Hence, from (J23SD 
we get 

+ 11^(0 IIl»(R)<£- (2-41) 
Noticing that f 1 2 . 2 [) i and (12.2^ -? can be rewritten as 

=(^ + \e) <P® - (2.42) 
\p J p 

and 

0"(O = f ^ + Kl 2 ) 5(0 - (2.43) 
\ p J p 

we apply ( gap to f[2~4^j) and (P233D to obtain 

||p"(0IU»(R) + F'(0 ||l*(r)<c. (2.44) 
Using (j2~4"ll and fT2T44|) . analogously to (12341) . we infer that 

W(O\\L>(M) + \\0'(t;)\\L*(n)<c. (2.45) 

Putting (I2.41I) - (I2.45I) together, we immediately obtain (I2.34p . This completes the proof. □ 
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2.3. Exponential growth rate 

In this subsection we use the Fourier synthesis to build growing solutions to fll.8p out of the 
solutions constructed in the previous subsection (Theorem 12.21) for any fixed spatial frequency 
(Gf 2 with |£| > 0. The constructed solutions will grow in-time in the Sobolev space of order k. 

Theorem 2.3. Let < R\ < R2 < 00 and f G C^(Ri, R2) be a real-valued function. For £ G M 2 
with |£| G (0, 00), define 

v(£ x 3 ) = -i<p(£, x 3 )e 1 - i6{£, x 3 )e 2 + ip(£, x 3 )e 3 , 

where (</>, ■?/>, 7r)(£, x 3 ) with A(|£|) > is the solution given by Theorem \2.2\ Denote 

e (f,x) = -^/ fmMZ,*3)e« m e«tdZ, (2.46) 
4vr 2 J R 2 

u(t,x) = -L / A(0/(|el)v(e,x 3 )e A d^V^de, (2.47) 
4vr 2 J M 2 

g(*,x) = -L / A(0/(|£lM£,* 3 )e A ^'V^d£, (2.48) 
4vr z J R 2 

Then, (g, u, g) is a real-valued solution to the linearized problem $1.81) along with \1.J\) . For every 



k G N, we have the estimate 

\\g(0)\\m + HO)!!** + ||g(0)|U* < D k (J (1 + |£| 2 ) fe+2 |/(lel)| 2 d£) < 00, (2.49) 

where D k > is a constant depending on k, p, R\, R2 and g. Moreover, for every t > we have 
(g(t),u(t),q(t)) G H k , and 

e tXoiJ) \W)\\m < Mt)\\n- < e tA \\g(0)\\^, (2.50) 

e tXo(f) ||u<(0) \\ h m < \\ui(t)\\ Hk <e tA \\ Ul (0)\\HK, 2 = 1,2,3, (2.51) 

e* Ao(/) ||g(0)||* fc < \\q(t)\\ Hk < e* A ||g(0)||^, (2.52) 



where 



and 



A (/) = inf A(|e|)>0 (2.53) 

I5|esupp(/) 



A= sup \(\£\)<y/g VP'/P ■ (2-54) 

0<|£|<+oo 

In particular, 

||u 3 (0)|| Hfc >0 iff^O, (2.55) 
and we can further take proper constants R±, R2, such that 

A (/) = A/2. (2.56) 

Proof. Obviously, f !2.53j) . (I2.54p and (I2.56j) follow from Proposition |23J For each fixed £ G M 2 , 

^,x) = -p7(|£|)^ 3 (£,x 3 )e A ^)V^, 
U (t,x) = A(|£|)/(|£|)v(£,a; 3 )e A ^)V^, 
g(t,x) = A(|£|)/(|£|) 7 r(£,x 3 )e A ^V^ 
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give a solution to ( II. 8p . Since / G C^°(Ri, R 2 ), Lemma I2TTI implies that 

sup \\d^(g, u, q)(£, < oo for all k G N. 

£esupp(/) 

These bounds show that the Fourier synthesis of the solution given by (I2.46p - (l2.48p is also a 
solution of (11. 8p . Because / is real- valued and radial, by Remark 12.21 we can easily verify that 
the Fourier synthesis is real-valued. 

The estimates (I2.49P and (I2.55P follow from Lemma 12.11 with arbitrary k > 0, and the fact 
that / is compactly supported and 



/ (i + iei 2 ) fe - j '|^,z 3 )rdedx 3 



3=0 
k 



E / (1 + \Z\ 2 ) k - J -)\\l, m d£ for r = g, or u h or q. 



(2.57) 



j=0 

Finally, we can use f l233|) . fl2T54"P and ( 1237]) to obtain the estimates f l230|) -f l232|) . □ 



3. Uniqueness of the linearized equations 

In this section, we will show the uniqueness of solutions to the linearized problem, which will 
be used in the proof of Theorem 11.11 in Section 5. We first define the function space of strong 
solutions. 

Q{T):={{Q,u,q) \geC°([0,T},L 2 ), Vq E L 2 (0, T; ff£ c ), 

u e C°([0,T],{L 2 ) 3 ) n L 2 {0,T;{H 2 ) 3 ), d t u G (L 2 ((0,T) x R 3 )) 3 }. 

We claim that the solution to the linearized problem is unique in the function space Q. 

Theorem 3.1. (Uniqueness) Assume that (g, u, q), (g,u,q) G Q{T) are two strong solutions of 
M.8\) with (g, u)(0) = (g, u)(0). Then, (g,u,Vq) = (g,u,Vq). 

PROOF. Let (g, u,q) = (g — g, u — u, q — q). Then (g, u, q) G Q{T) is still a strong solution to 
the linearized system ( II. 8p with (g, u)(0) = (0,0). 

Multiplying (OD2 by ip G (C%°((0,t) x M 3 )) 3 with divcp = 0, integrating over (0,t) x M 3 , and 
then integrating by parts, we arrive at 

pd t u ■ ipdxdr + / /iVu : V^dxdr = — g / / p^ 3 dxdr. 
Jo Jr 3 Jo Jr 3 



Using the standard density argument (see, for instance, [4j or [17|, Section 2.1]), we could take u 
as a test function to get 



/ / pd t u ■ udxdr + / / /iVu : Vudxdr = — g / / ^w 3 dxdr, (3.1) 
Jo Jr 3 Jo Jr. 3 Jo Jr 3 

which will be employed to derive an energy-like estimate. In fact, recalling 

u G L 2 (0, T; (if 1 ) 3 ) n C°([0, T], (L 2 ) 3 ) and d t u G (L 2 ((0, T) x M 3 )) 3 , 
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(3.2) 



and u(0) = 0, we easily deduce 

/ I pd t u ■ udxdr = — / pu 2 (t)dx. 
Jo Jn 2 Jq 

Since g e C°([0, T],L 2 ) and g(0) = 0, the equation (|TB)» i gives 

£>(£, x) = / p'u 3 (s, x)ds for any £ > 0, 
Jo 

Consequently, with the help of the regularity of <9jW3, the property of absolutely continuous 
functions and Fubini's theorem, we conclude that 



£>-u 3 dxdr 



J M 3 
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>/ 



ti 3 (s, x)dsM 3 (r, x)drdx 



// / — ( / //;;(. s.x)d.s j drdx 



lL p iIo u3iT ^ dT ) dx 



(3.3) 



<- 
~2 



p'w^dxdr 



< 



L°°(M) JO 



P 



u 2 dxdr. 



Substituting (I3.2p and (I3.3I) into f)3.ip . we conclude that 



pu 2 (t)dx 



/iVu : Vudxdr < - 

o Jr 3 2 



L°°(R3) JO 



pu 2 dxdr, 



L°°(M) JO 



/ ||v/pu|| L 2dr. 
Jo 



(3.4) 



which yields 

\\VP<t)\\h<T 

Applying Grownwall's inequality to (I3.4p . we get 

||Vpu(t)|| 2 2 =0 for any te [0,T], 
which yields u = 0, i.e., u = u, since p > 0. This, combined with (jl.8P i and (I1.8P ?. proves that 

(p, u, Vg) = (p, u, Vg) for any t £ (0, T]. 
Thus, the desired conclusion follows. □ 



4. Nonlinear energy estimates of the perturbed problem 

In this section, we shall derive some nonlinear energy estimates for the perturbed problem, 
which will also be used in the proof of Theorem II. II in Section 5. To this end, let (g, u, g) be a 
classical solution of the perturbed problem (jl.5p - (ll.7p with p := g + p > in [0, T] x M 3 for some 
T > 0. Moreover, we assume that the classical solution (g, u, q) satisfies the initial condition 

\J\\Qo\\ 2 H i + Kl^ =S <1. (4.1) 
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The restricted relation between T and 5q will be given at the end of Subsection 4.1. 

In what follows, we denote by Cj (i = 1,2, •••), C{T) and C generic positive constants 
depending on p, g and p. In addition, C(T) also depends on T and is nondecreasing with respect 
to T; the subscript i of C, emphases that we may repeat to use the constant Cj in the process of 
estimates. 

4-1. Estimates for \\u t \\ L 2 and ||Vu||#-i 

We first observe that the continuity equation (ll.50 i and the incompressibility condition ( II. 50 ^ 
imply immediately that for any t e (0, T], 

a := inf {p (x)} < p(t) < sup{p (x)} := p or a - p < g(t) < p - p, (4.2) 



and 

A r p'q 

\\y/pu\\ L 2 < 2a" 1 ||p / || L oo ||^|| L2 H^/pull^. (4.3) 



7 p 

tMWWI* = ~ 2 / P>sdx<2 
at j R 3 



VP 



L-' 



Multiplying ( 11.50 9 by u, using (11.50 1 , and then integrating (by parts) over (0, t) x M 3 , we 
obtain 

p|u| 2 (t)dx + p I |Vu| 2 dx = — g I gu^dx.. 



2dt 

Since the integral on the right-hand side is bounded from above by <7a - 5||0|| L a||y7>u||£2, we get 

j t \\V~P^)\\h + Mil Vu||| 2 < 2ga~* \\g\\ L2 \\VMv- ( 4 - 4 ) 
Combining (14.30 with ( 14.40 and using Cauchy-Schwarz's inequality, we obtain 

j t (Mt)\\h + HvWOIliO + Hlvu|| 2 L2 <Ci(\m\\h + ( 4 - 5 ) 

which implies 

\\9(t)\\h + \\V~P<t)\\h<Soe CV . (4.6) 
In particular, making use of ( 14.20 . ( 14.50 and ( 14.60 . we arrive at 



\Q(t)\\h + \Ht)\\h 



+ f ||Vu(s)||£ 2 dr < C5 e Clt . (4.7) 
Jo 



To control u t , we multiply (11.50 9 by u t in L 2 and apply Cauchy-Schwarz's inequality to infer 
that 

-IIVMll!^ ^l|Vu(t)||| a <C(|| e ||| a + || v ^u.Vu||i 2 ). (4.8) 

To bound the second term on the right-hand side of ( 14.80 . we recall that (u, q) is a solution of 
the Stokes equations: 

— pAu + Vg = — pu t — p(u • Vu) — gge^, div u = in M 3 . 

Therefore, the classical regularity theory on the Stokes equations (see |8|, Theorem 2.1]) gives 

||V 2 u||| 2 <C 2 (\\Vpu t \\ 2 L2 + ||Vpu • Vu|| 2 2 + g\\ e \\ 2 L2 ), 
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which combined with (14. 8p results in 

\\\Vput\\h + ^l|V 2 u||l 2 + ^||Vu(t)||| 2 < CsUh + H u ■ Vu||| 2 ) with C 3 > 1. (4.9) 
If we apply Holder's and Sobolev's inequalities, we have 

||U - VuHia <||u||i 6 ||Vu||i 3 < ||u||| 6 ||Vu|| i2 ||Vu|U6 

<Co||Vu||i a ||V U || H i < ^^||Vu||l 2 + -^||Vu||^. 
Substituting the above inequality into (14. 9 p and integrating over (0,t), we conclude that 

+ y HV 2 u||i 2 + /i|||Vu(t)||i 2 < C(\\ e \\b + HVu||| 2 ). (4.10) 
Letting 5 e ClT < 1, we get from (fl~5j) and (j4~9l) that 

n\\Vu(t)\\l 2 < / < C(||^||| 2 + ||Vu||l 2 )d S + / ,||Vuo||| 2 
Jo 

<C5 2 (e c ^ - 1) + C f || Vul^da + /x|| Vu ||i 2 (4.11) 
Jo 

which yields 



Now, we take 



» Vu » £ < 412) 



,1,1 i 

1 = mm < ——In- 



Ci 5o'45 2 C 4 3 J- 

In particular, there exists a sufficiently small constant 5\ > 0, such that 

1 r 

T = ^-^ for any 5 G (0,50. (4-13) 
From now on, we always assume that 5q and T satisfy the relation Thus, f )4.12p gives 



||Vu(t)||| 2 < V /25 2 C| < C5 for any t e (0,7]. (4.14) 
Making use of (g^D, (jMD, ffl~T4]) and the first inequality in (j£TTj) . we deduce from fl^TTUD that 

A||u t (5)|| 2 L2 + ||Vu( S )|| 2 , 1 )d S < C(7)5l for any f e (0,T]. (4.15) 
Jo 



la + HO ||^ 
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4-2. Estimates for WVu t \\ L 2 and ||V 2 u|| i e 

Using f ll.5P i and keeping in mind that p = q + p, we can rewrite ( II. 5p ? as 

pu t + pu • Vu + Vp = pAu - pge 3 , 

whence, by taking the time derivative, 

pu tt + pu • Vu t - pAu t + Vpt = -pt(u t + u • Vu + ge 3 ) - pu t ■ Vu, 

which, by using the continuity equation, can be written as 

P Ql u <! 2 ) + pu ■ V Ql u t| 2 ) - P Au i • u t + Vpt • u f 
= div(pu)(u t + u • Vu + ge 3 ) ■ u t - p(u t • Vu)u ( . 
Hence, by integrating by parts, we see that 

™ / p|u t (t)| 2 dx + p f |Vut| 2 dx 

1 at JR3 J R 3 

J 2p|u||ut||Vut| +p|u||ut||Vu| 2 + p|u| 2 |ut||V 2 u| (A.IQ) 



< 



+ p|u| 2 |Vu||Vut| +p|ut| 2 |Vu| + (7p|u||Vut| -=J2 l 



i=l 



where Ij can be bounded as follows, employing straightforward calculations. 

h < 2||p||^||u|| i 6||^put|| L 3||Vut|| L 2 < 2||p||} i / i||u|| L 6|| v /put||J j / 2 2 || v /put||^ / 6 2 ||Vut|| L 2 

< Cllpll^HVulUallVputll^llVutll 3 /, 2 < C(,)||Vu||l 2 ||yput||i 2 +,||Vut||| 2 , 
h < CllplUccllVull^HVutll^llVulliallVullHi < C^llVull^HVull^ +£||Vut||| 2 , 
h < C||p|Uo ||Vu||! 2 ||Vut|U 2 ||V 2 u|U 2 < C( £ )||Vu||l 2 ||V 2 u||i 2 +e||Vut||i 2 , 
h < C||p|| I/ o ||Vu||| 2 ||Vu|| H i||Vut|U 2 < C(e)\\Vu\\t 2 \\Vu\\ 2 H1 +s\\Vu t \\ L 2, 
h < Cllpll^llVulU.IIVputll^llVutllK 2 < C( £ )||Vu||| 2 ||Vput||i 2 +£||Vut||| 2 , 

/6<C(£)||u|| 2 2 +£||VUt|| 2 2 , 

where C(e) is a positive constant which may depend on e. Inserting all the above estimates into 
( I4.16p . we conclude 

d \\VpMt)\\h + livut|| 2 2 < cf||Vu||| 2 (||Vput||! 2 + ||vu||^) + || U ||| 2 



dt 

which, by integrating over (r, t) and using (I4.15p . leads to 

-t 



\\Vpn t (t)\\h + I l|Vut( S )||| 2 d S < ||Vput(r)|| 2 2 +C(T)5 2 . (4.17) 
On the other hand, multiplying f 1 1 . 5 1) q» by u t in L 2 and recalling divu t = 0, we find that 
/ p|u f (t)| 2 dx = / (-gge 3 - pu • Vu + pAu - Vg) • u t dx 

J®? J R 3 

—gge 3 — pu • Vu + pAu) ■ u t dx, 
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whence, 

/ p|u t (t)| 2 dx< C [ (g 2 + |u| 2 |Vu| 2 + |Au| 2 )(t)dx. 
Taking t — > in the above inequality and using ( 14. ip . one gets 

limsup / p|u t (t)| 2 dx < C5q. 

Therefore, letting r — > in (I4.17p . we conclude that 

\\VpMt)\\h + f l|Vu t ( S )||| 2 d5 < C(T)6 2 . (4.18) 
Jo 

To derive estimates of higher derivatives, we recall again that the pair (u, q) solves the Stokes 
equations: 

—pAu + Vq = —pu t — p(u ■ Vu) — ggez, divu = in R 3 . 
It follows from the classical regularity theory for Stokes equations that 

|| V 2 u||i 2 + || Vq\\\ 2 <C\\ - pu t - p(u • Vu) - gge 3 \\ 2 L2 

<C(||pu t ||| 2 + ||pu- Vu||| 2 + \\g\\ 2 L2 ) 

<C(\\p\M\y/pUt\\h + IIPlli-l|u||£e||Vu|||3+^|b|| 2 i2 ) ( 4 - 19 ) 

<c(HVput||! 2 + ||vu||| 2 + \\ g \\ 2 L2 ) + ^iivuii^. 

Putting the estimate (14. 15j) . (14.181) and (14. 19j) together, we obtain 

||V 2 u(t)||i 2 + \\Vq(t)\\ 2 L2 < C{T)5l for any t E (0,T]. (4.20) 
Arguing analogously to ( I4.3H . we can deduce from (ll.8l) i that 

^lk(*)IU» <6||jS'|Uco||u(t)|U8 for any t G (0, T\. (4.21) 

By virtue of the estimates (14TB . fTCTol) . fj4~T8l) . (I4T201) and (S2U), and the fact that 
|| V 2 u||i 6 + ||Vg||£ 6 <C(||pu t ||| 6 + ||pu ■ Vu||| 6 + g\\g\\ 2 L e), 

we deduce that 

/ (l|Vu||^, 6 + ||Vg||i» + ||Vu||£„)( a )d* < C(T)5 2 . (4.22) 
Jo 

4-3. Estimates for \\g\\m 

Observing that each g satisfies 

{\Q Xj \ 2 )t + div(|& 3 | 2 u) = -2g Xj ((yp-u) Xj + Vg-u Xj ), 

we integrate over R 3 , sum over j and use fll.2[) to infer that 

| f |Vf?(t)| 2 dx <C [ (\Vu\\Vg\ 2 + (|u| + |Vu|) |V^|dx 

at JR3 Jm? 

<C(||Vu|| L oo||V^|U 2 + ||u||^i)||Ve|| La . 
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Hence, 

||V^)|| L2 <e c /ollVu(s)|| LOO d s ^iv^H^+C^llu^ll^d^ . (4.23) 

From (H77j) . fl422|) and pg| it follows that 

\\g(t)\\ m <C(T)5 for any te (0,T]. 
Summing up the above estimates, we arrive at the following property: 

Proposition 4.1. There exists a 5\ G (0, 1], sitcA t/iat a/II^oII/p + [juojj^ = ^ E (0, 5i), then 
any classical solution (g,u,q) to U.5\) . emanating from the initial data (go,^o), satisfies 

sup (b||^ + ||u||^ + ||u t ||i 2 + ||Vg||i 2 )W 

0<s<T s 
"T S 



+ / *(HVu||^ + ||Vu t ||| 2 )( s )d S < C(T S )5 2 , 
Jo 



where T§ = min{T max , —C\ 1 ln5 } and T max denotes the maximal time of existence of the classical 
solution (g, u, q). 

Remark 4.1. The local existence of classical solutions and global existence of classical small so- 
lutions to the 3D nonhomogeneous incompressible Navier-Stokes equations have been established 



by many authors, see |22|, |25| for example. In particular, by a slight modification, one can follow 
the proof of 22, Theorem 1.1] and use the expanding domain technique in 0, ls[ to obtain a 
local existence result of classical solutions to the Rayleigh- Taylor instability problem (ll.5p -f Tl~Tj) 
defined on (0,T max ) x M 3 , where T max -» oo as the initial data ||^o||if 3 (R 3 ) + ||uo||j3-4(R3) ->■ 0. The 



proof is standard by means of energy estimates, and hence we omit it here. 

5. Proof of Theorem 11.11 

In this section we start to show Theorem 11.11 To this end, we first construct a linear solution, 
and a family of nonlinear solutions which posses some special properties. Then, by contradiction 
argument, we show that there exists a nonlinearly unstable solution satisfying the properties as 
stated in Theorem 11.11 Suppose that s > 2, 5 > 0, K > 0, and F satisfying (II. 9p . are arbitrary 
but given. 

5.1. Construction of a solution to the linearized problem 

In view of Theorem 12.3} we can construct a classical solution (g, u, q) to fll.6p - fll.8p satisfies 
the properties fl2.50p -f l2.52p . (I2.55P and (12.561) . Noticing that the solution (g, u) is independent 
of s and ||w3(0)||# s > 0, we define 

, n S{g,u,q) 

\\(e, u)(o)||jf» 

and find that (p, v, p) is still a classical solution to (jl.6p — (11.8p satisfying all the properties of 
(g,u,q). Moreover, 

\\(p,v){0)\\ B . = 6. 

Now, we denote 

. f s Kjjg lfe(Q)lb 

i := i (s) := T --— — = < 1, 

(0,u)(O) H s 5 



then i > 0. Consequently, defining t K = fln^p and recalling that t> 3 satisfies (I2.5ip . we obtain 

\\vz{tK)\\L*>e tKA/2 i 5>2K5. (5.2) 
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5.2. Construction of a solution to the corresponding nonlinear problem 

Based on the initial data (p, v)(0) of the solution (p, v,p) given in (15. ip . we proceed to 
construct a family of solutions to the perturbed nonlinear problem. Define 

{qI%) :=e(p,v)(0) for eE (0,1). 

Noticing that 

(&u§) e (H°°)\ and < Ss < 5, (5.3) 

we see by Remark 14.11 and Proposition 14.11 that there exists a constant e%, such that for any 
e G (0, £i), there exists a classical solution (g £ ,u £ ,q £ ) to the nonlinear RT problem f 1 1 . 5 [) — ( TTTTj) 
on (0,T e max ) x M 3 with T e max > t K , satisfying 

sup W(t)\\* + ||u £ (t)||i 2 + ||uf(t)||| 2 + \\W(t)\\h) 

0<t<T K 

rT K (5.4) 



+ / A (HVu £ ( s )||^ + \\Vu £ t (s)\\h)ds < C(T k )6 2 e 2 , 
Jo 



where C{Tk) is independent of e. In addition, 

sup || (ff + p)(t)||z,°° < C{5) for some constatnt C(5) depneding S, 

0<t ~ TK ' (5 5) 

SU P Wf II W) < C(T K ,tt')5 2 e 2 for any O' CC M 3 . 

0<t<T K 

Obviously, to complete the proof of Theorem IX . X ^ it suffices to show the following lemma. 

Lemma 5.1. There exists an Eq G (0, Ex), such that the classical solution (g £ °, u £ °) of M.5\) - §T7fy , 
emanating from the initial data (^q°,Uq°) ; satisfies 



2 2K 

\ul°(t K )\\ L2 > F(\\(qI°,u £ )\\ H s) forsomet K e ( 0, -In 



C(0, TJT), 



A i 

where T™ ax denotes the maximum time of existence to the solution (g £0 ,u £0 ). 

Proof. We shall show the lemma by contradiction. Suppose that for any e G (0, E\), the classical 
solution (g £ , u £ , q £ ), emanating from the initial data (^q, Uq), satisfies 

M(t)\\v < n\m,vl)\\H°) for any t G (0,t K \ C (0,T e max ), 
which, together with (15.31) and (II. 9p . yields 

\\u £ 3 (t)\\ L2 < K\\(g £ Q ,v £ )\\ H s < K5e, Vt G (0,T £ max ). (5.6) 
We denote (g £ , u £ , q £ ) := (g £ , u £ , q £ )/£, then they satisfy 
~g\ + u £ ■ V(Eg £ + p) = 0, 

(Eg 6 + p)uf + e(eg £ + p)u £ ■ Vu £ + V<f + gg £ e 3 = pAu £ , (5.7) 
divu e = 0. 

with initial data 

(^,u £ )(0) = (p,v)(0). (5.8) 



22 



Moreover, the following estimates hold because of fl5.4p - fl5.6p . 

sup \\(eg £ + p)(t)\\ L »o <C(8), sup \\ul(t)\\v < K5, (5.9) 

0<t<T K 0<t<T K 

sup \\<f \\ 2 mm < C(T K , n')5 2 for any SI' CC M 3 , (5.10) 

0<t<T K 

sup (ii^n^ + iiu £ ii^ 2 + ii^ii| 2 + iivrn| 2 )(t) 

0<t<T K 

+ [ TK {\\V&\\h + HVuI||| 2 )(s)d S < C(T K )S 2 . (5.11) 
Jo 

The continuity equation fll.8P i combined with f l5.9p -f l5~TT|) immediately implies 

sup \\g e t \\ 2 L 2 <C(T K )S 2 . r 5U ) 

0<t<T K v ' 

Thus, from fl5.9p -( l5Tl~2"l) we immediately infer that there exists a subsequence (not relabeled) 
of {(g e , u e , q e )}, such that 

(ft 5 , u£, <f ) ->■ (ft, u t , g) weakly-star in L°°(0, T K ; (L 2 ) 4 x iZjJJ, 
(£ e ,u £ ) -> (ftu) weakly-star in L°°(0,T K ; H 1 ) x (if 2 ) 3 ), 
(^, u £ ) -> (ft u) strongly in C°([0, TV], (L 

and 



2 

locJ J' 



sup ||«3(t)||^ < K8, (ftii) G C°([0,T^],(L 2 ) 4 ). (5 13) 

If one takes to the limit as e — > in the equations (15. 7p . one gets 

d t g + u ■ Vp = 0, 
p<%u + Vg + #£e 3 = pAu, 
divu = 0. 

Therefore, we see that (ft u) is just a strong solution of the linearized problem (jl.6|) - (ll.8|) . Of 
course, (p, v) is also a strong solution of (11.61) — ( 1X781) . Moreover, (ftii)(0) = (p, v)(0). Hence, 
according to Theorem 13.11 

v = u on [0,T K ] x R 3 . 
Thus, we may chain together f)5.2p and the inequality ( 15 . 1 3 j) to get 

2Kb < \\va(tK)\\iJ>ra*) < llusC**)!!^ 3 ) - K ^ 



which is a contraction. This completes the proof of Lemma 15. 1[ and hence the proof of Theorem 

EU □ 
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